ARROW-11559: [C++] Use smarter Flatbuffers verification parameters #9447
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Note that we could adopt a similar strategy when validating the Parquet Thrift structures (related: https://issues.apache.org/jira/browse/PARQUET-1877). |
|
cc @emkornfield |
pitrou
force-pushed
the
ARROW-11559-fbb-verification-params
branch
2 times, most recently
from
bad3f4c to
b485e01
Compare
Flatbuffers is able to encode a virtually unbounded of schema fields in a small buffer size. Verifying that many fields with the Flatbuffers verifier seems to result in potentially unbounded verification times, which is a denial of service risk. To mitigate the risk, impose that a Flatbuffers buffer cannot represent one more than one Flatbuffers table per buffer bit, which should always be true for well-formed Arrow IPC metadata. Indeed, the only recursive table, the `Field` table in Schema.fbs, mandates the presence of its `type` member (though it's not marked as required in the Flatbuffers definition, it's validated by the IPC read routines).
pitrou
force-pushed
the
ARROW-11559-fbb-verification-params
branch
from
b485e01 to
b3f1e6f
Compare
|
To quote an answer that I got on the Flatbuffers mailing-list: a buffer of a given size can basically encode a nearly unlimited number of tables, since Flatbuffers can deduplicate tables (e.g. you can have a Field with N times the same child, itself with M times the same child, etc., which can produce a combinatory explosion). |
|
Hmm, I'll add the regression file as a separate PR then. |
nevi-me
pushed a commit
to nevi-me/arrow
that referenced
this pull request
Feb 13, 2021
Flatbuffers is able to encode a virtually unbounded of schema fields in a small buffer size. Verifying that many fields with the Flatbuffers verifier seems to result in potentially unbounded verification times, which is a denial of service risk. To mitigate the risk, impose that a Flatbuffers buffer cannot represent one more than one Flatbuffers table per buffer bit, which should always be true for well-formed Arrow IPC metadata. Indeed, the only recursive table, the `Field` table in Schema.fbs, mandates the presence of its `type` member (though it's not marked as required in the Flatbuffers definition, it's validated by the IPC read routines). TODO: * [ ] Add OSS-Fuzz regression file Closes apache#9447 from pitrou/ARROW-11559-fbb-verification-params Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Micah Kornfield <[email protected]>
sgnkc
pushed a commit
to sgnkc/arrow
that referenced
this pull request
Feb 17, 2021
Flatbuffers is able to encode a virtually unbounded of schema fields in a small buffer size. Verifying that many fields with the Flatbuffers verifier seems to result in potentially unbounded verification times, which is a denial of service risk. To mitigate the risk, impose that a Flatbuffers buffer cannot represent one more than one Flatbuffers table per buffer bit, which should always be true for well-formed Arrow IPC metadata. Indeed, the only recursive table, the `Field` table in Schema.fbs, mandates the presence of its `type` member (though it's not marked as required in the Flatbuffers definition, it's validated by the IPC read routines). TODO: * [ ] Add OSS-Fuzz regression file Closes apache#9447 from pitrou/ARROW-11559-fbb-verification-params Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Micah Kornfield <[email protected]>
GeorgeAp
pushed a commit
to sirensolutions/arrow
that referenced
this pull request
Jun 7, 2021
Flatbuffers is able to encode a virtually unbounded of schema fields in a small buffer size. Verifying that many fields with the Flatbuffers verifier seems to result in potentially unbounded verification times, which is a denial of service risk. To mitigate the risk, impose that a Flatbuffers buffer cannot represent one more than one Flatbuffers table per buffer bit, which should always be true for well-formed Arrow IPC metadata. Indeed, the only recursive table, the `Field` table in Schema.fbs, mandates the presence of its `type` member (though it's not marked as required in the Flatbuffers definition, it's validated by the IPC read routines). TODO: * [ ] Add OSS-Fuzz regression file Closes apache#9447 from pitrou/ARROW-11559-fbb-verification-params Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Micah Kornfield <[email protected]>
michalursa
pushed a commit
to michalursa/arrow
that referenced
this pull request
Jun 13, 2021
Flatbuffers is able to encode a virtually unbounded of schema fields in a small buffer size. Verifying that many fields with the Flatbuffers verifier seems to result in potentially unbounded verification times, which is a denial of service risk. To mitigate the risk, impose that a Flatbuffers buffer cannot represent one more than one Flatbuffers table per buffer bit, which should always be true for well-formed Arrow IPC metadata. Indeed, the only recursive table, the `Field` table in Schema.fbs, mandates the presence of its `type` member (though it's not marked as required in the Flatbuffers definition, it's validated by the IPC read routines). TODO: * [ ] Add OSS-Fuzz regression file Closes apache#9447 from pitrou/ARROW-11559-fbb-verification-params Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Micah Kornfield <[email protected]>
This was referenced Jul 26, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Flatbuffers is able to encode a virtually unbounded of schema fields in a small buffer size.
Verifying that many fields with the Flatbuffers verifier seems to result in potentially unbounded verification times, which is a denial of service risk.
To mitigate the risk, impose that a Flatbuffers buffer cannot represent one more than one Flatbuffers table per buffer bit, which should always be true for well-formed Arrow IPC metadata. Indeed, the only recursive table, the
Fieldtable in Schema.fbs, mandates the presence of itstypemember (though it's not marked as required in the Flatbuffers definition, it's validated by the IPC read routines).TODO: